Two SVDs produce more focal deep learning representations

A key characteristic of work on deep learning and neural networks in general is that it relies on representations of the input that support generalization, robust inference, domain adaptation and other desirable functionalities. Much recent progress in the field has focused on efficient and effective methods for computing representations. In this paper, we propose an alternative method that is more efficient than prior work and produces representations that have a property we call focality – a property we hypothesize to be important for neural network representations. The method consists of a simple application of two consecutive SVDs and is inspired by (Anandkumar et al., 2012). In this paper, we propose to generate representations for deep learning by two consecutive applications of singular value decomposition (SVD). In a setup inspired by (Anandkumar et al., 2012), the first SVD is intended for denoising. The second SVD rotates the representation to increase what we call focality. In this initial study, we do not evaluate the representations in an application. Instead we employ diagnostic measures that may be useful in their own right to evaluate the quality of representations independent of an application. We use the following terminology. SVD (resp. SVD) refers to the method using one (resp. two) applications of SVD; 1LAYER (resp. 2LAYER) corresponds to a single-hidden-layer (resp. twohidden-layer) architecture. In Section 1, we introduce the two methods SVD and SVD and show that SVD generates better (in a sense to be defined below) representations than SVD. In Section 2, we compare 1LAYER and 2LAYER SVD representations and show that 2LAYER representations are better. Section 3 discusses the results. We present our conclusions in Section 4.